Orsanco notes

1. The 10 year history file and the 2 year data gap files are not on the FTP Server. Instead they are on Google Drive
   1. 10 year https://drive.google.com/file/d/1aRjEmQDxQDidPG5gFFd1ihmyN0Wz8TM2/view?ts=63c1634f
   2. 2 year gap data https://drive.google.com/file/d/1Ze01ztP6cimFvem6wabBetl9Ban2BDo0/view?ts=63e167e6
2. The history file never gets downloaded from the FTP server It is always Local therefore we do not access the FTP Server.
3. How the file scripts work together. The scripts should be run in a specific order.
   1. The scripts for this project have to be run in a specific order. Some are explicitly called and some call others. Here is the order they should be called on a clean database.
      1. Setup_cross_sections (Called explicitly) It populates the River, Reach, and Station tables.
      2. insert_flow_history (Called explicitly) It populates the flow history table based on the (No Upsert so make sure you don’t have conflicts in the file.
         1. There have been modifications involving csv files to this I am not 100% up to date on this I was planning on using the Flow Diagram to figure it out today. Update Flow diagram does not really help. Nothing was changed othe
      3. insert_flow_daily_history(Called explicitly) It takes the database as is and calculates the daily history data (No Upsert so only run this after all history data is done)
      4. download_hdf_ftp (Called explicitly) This pull data down from the FTP Server unzips the files and calls update_flow to insert into the database. Then it note the file in the processed files table in the database.
      5. update_flow (Called by download_hdf_ftp) Upserts the data into the database it upserts 1st into the riverflow flow table then calculates and upserts into the dailyriverflow table. This is the only upsert but it does take longer.
4. As of Today 2/9/2023 I have downloaded all the files from the FTP Sever using WinSCP and have placed them on the Micro Machine at the following Location. C:\Project\ORSANCO\From NOAA\All files from FTP Server on 2_9. I am expecting Brian to remove the following files and replace them with other January files at some point today. The rest of the files will likely stay the same.
   1. OhioHECRas_output_20230101.tgz
   2. OhioHECRas_output_20230105.tgz
   3. OhioHECRas_output_20230110.tgz
5. Because the files for the Data gap will be local we will need to do the following:
6. There is a local path where the files are saved when they are downloaded from the FTP Sever. This value is saved in local settings. Local Settings override settings (although there is no setting for this in settings which is a bug I will fix.) On my machine only this is set to my test file for ORSANCO. C:\test\ORSANCO. Having a test file on the C drive is a GQC standard set by Jake. The reason I set it at this location is that this way I could use the files for testing without having to go to the server and download. So it was convent.
7. We do have a table that tracks what FTP files have been imported. It is called flowfileprocessed. This table is currently only referenced in the download_hdf_ftp.py file therefore if we do not use this process the data will not be entered into the database. This means that the 10 year the 2 year gap data and the files from the beginning of January will not be entered into the database unless we modify code.
8. Note We are going to have to create a new script that will unzip a local file without calling the FTP sever. It should do the following:
   1. Point at a specific folder.
   2. Get all the TGZ files in that folder.
      1. Create a list.
      2. Unzip the folders and upsert into the database.